ML-Regression

Linear Regression is used for regression tasks to predict continuous outputs by minimizing the mean squared error. Ridge Regression is a regularized version of linear regression that adds a penalty term to the loss function to prevent overfitting. Logistic Regression is used for binary classification tasks to predict probabilities and classify data points.

Linear Regression

1. Hypothesis

Linear regression models the output $y$ as a linear function of the input features $x$:

$$h_{\theta }(x)={\theta }^{T}x={\theta }_0+{\theta }_1{x}_1+\cdots +{\theta }_n{x}_n$$

2. Cost Function

The cost function used is the Mean Squared Error (MSE):

$$J(\theta )=\frac{1}{2m}\sum _{i=1}^m{(h_{\theta }(x^{(i)})-y^{(i)})}^2$$

3. Optimization

• Gradient Descent:$${\theta }_j:={\theta }_j-\alpha \frac{1}{m}\sum _{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)})x_j^{(i)}$$
• Normal Equation:$$\theta ={\left(X^{T}X\right)}^{-1}{X}^{T}y$$

---

Logistic Regression

1. Hypothesis

Logistic regression maps the linear function to a probability using the sigmoid function:

$$h_{\theta }(x)=g({\theta }^{T}x),g(z)=\frac{1}{1+e^{-z}}$$

• Prediction Rule:
  • Predict $y=1$ if $h_{\theta }(x)\ge 0.5$.
  • Predict $y=0$ if $h_{\theta }(x)<0.5$.

2. Cost Function

The cost function for logistic regression is:

$$J(\theta )=\frac{1}{m}\sum _{i=1}^m[-y^{(i)}\log (h_{\theta }(x^{(i)}))-(1-y^{(i)})\log (1-h_{\theta }(x^{(i)}))]$$

3. Optimization

• Gradient Descent:$${\theta }_j:={\theta }_j-\alpha \frac{1}{m}\sum _{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)})x_j^{(i)}$$

4. Sigmoid Function Properties

• $g(z)$ maps inputs to $[0,1]$, representing probabilities.
• Derivative:$$g^{\prime }(z)=g(z)(1-g(z))$$

Ridge Regression

岭回归是一种线性回归的正则化变体，通过在损失函数中加入 $L2$ 正则化项来解决 多重共线性 和 过拟合 问题。

Loss function

岭回归的目标是最小化以下损失函数：

$$J(\theta )=\frac{1}{2m}\sum _{i=1}^m{(h_{\theta }(x^{(i)})-y^{(i)})}^2+\lambda \sum _{j=1}^n{\theta }_j^2$$

• $\lambda$: 正则化参数，控制正则化强度。
  • $\lambda =0$: 等价于普通的线性回归。
  • $\lambda \to \mathrm{\infty }$: 所有 ${\theta }_j\to 0$。
• $\sum _{j=1}^n{\theta }_j^2$: $L2$ 正则化项，用于约束参数大小，防止过拟合。

---

岭回归有一个闭式解，可以通过修改普通线性回归的正规方程得到：

$$\theta ={(X^{T}X+\lambda I)}^{-1}{X}^{T}y$$

• $X$: 设计矩阵。
• $I$: 单位矩阵。

梯度下降解 通过梯度下降最小化损失函数：

$${\theta }_j:={\theta }_j-\alpha (\frac{1}{m}\sum _{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)})x_j^{(i)}+2\lambda {\theta }_j)$$

• 正则化部分 $2\lambda {\theta }_j$ 是对 $\theta$ 的惩罚。

---

Logistic Regression

$$h_{\theta }=g({\theta }^{T}x)$$$$g(z)=\frac{1}{1+e^{-z}}$$

Suppose predict

"$y=1$" if $h_{\theta }(x)\ge 0.5$ $({\theta }^{T}x\ge 0)$ "$y=0$" if $h_{\theta }(x)<0.5$ $({\theta }^{T}x<0)$

$${\theta }^{T}x=0$$

Cost Function and gradient

The cost function for logistic regression is defined as:

$$Cost(h_{\theta }(x),y)=\{\begin{array}{ll}-\log (h_{\theta }(x))& \text{if }y=1\\ -\log (1-h_{\theta }(x))& \text{if }y=0\end{array}$$

When $y=1$:

• $Cost=0$ if $h_{\theta }(x)=1$.

• As $h_{\theta }(x)\to 0$, $Cost\to \mathrm{\infty }$.

• When $h_{\theta }(x)=0$, the model predicts $P(y=1|x;\theta )=0$, but $y=1$.

When $y=0$:

• $Cost=0$ if $h_{\theta }(x)=0$
• $Cost$ goes to infinite if $h_{\theta }(x)=1$

---

Simplification of Logistic Regression Cost Function

The overall cost function for logistic regression is defined as:

$$J(\theta )=\frac{1}{m}\sum _{i=1}^{m}Cost(h_{\theta }(x^{(i)}),y^{(i)})$$

Cost Function:

$$Cost(h_{\theta }(x),y)=\{\begin{array}{ll}-\log (h_{\theta }(x))& \text{if }y=1\\ -\log (1-h_{\theta }(x))& \text{if }y=0\end{array}$$

Note: $y$ is always 0 or 1.

$$J(\theta )=\frac{1}{m}\sum _{i=1}^m[-y^{(i)}\log (h_{\theta }(x^{(i)}))-(1-y^{(i)})\log (1-h_{\theta }(x^{(i)}))]$$

To fit parameters $\theta$: $\underset{\theta }{min}J(\theta )$

To make a prediction given new $x$:

Output $h_{\theta }(x)=\frac{1}{1+e^{-{\theta }^{T}x}}$

The cost function for logistic regression is:

$$J(\theta )=\frac{1}{m}\sum _{i=1}^m[-y^{(i)}\log (h_{\theta }(x^{(i)}))-(1-y^{(i)})\log (1-h_{\theta }(x^{(i)}))]$$

Gradient Descent Algorithm:

Repeat:

$${\theta }_j:={\theta }_j-\alpha \frac{\partial }{\partial {\theta }_j}J(\theta )$$$${\theta }_j:={\theta }_j-\alpha \frac{1}{m}\sum _{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)})x_j^{(i)}$$

(Simultaneously update all ${\theta }_j$).

$$g(z)=\frac{1}{1+e^{-z}}$$

Derivative:

$$g^{\prime }(z)=g(z)(1-g(z))$$$$h_{\theta }(x)=\frac{1}{1+e^{-{\theta }^{T}x}}$$